Distributed control plane for facilitating communication between a container orchestration platform and a distributed storage architecture

ABSTRACT

Techniques are provided for implementing a distributed control plane to facilitate communication between a container orchestration platform and a distributed storage architecture. The distributed storage architecture hosts worker nodes that manage distributed storage that can be made accessible to applications within the container orchestration platform through the distributed control plane. The distributed control plane includes control plane controllers that are each paired with a single worker node of the distributed storage architecture. Thus, the distributed control plane is configured to selectively route commands to control plane controllers that are paired with worker nodes that are current owners of objects targeted by the commands. In this way, the control plane controllers can facilitate communication and performance of commands between the applications of the container orchestration platform and the worker nodes of the distributed storage architecture.

TECHNICAL FIELD

Various embodiments of the present technology relate to a distributedcontrol plane. More specifically, some embodiments relate tofacilitating communication between a container orchestration platformand a distributed storage architecture using the distributed controlplane.

BACKGROUND

Historically, developers have built applications designed to be run on asingle platform. This makes resource allocation and program executionsimple and straight forward. For example, an application may be hostedon a server, and thus the application may utilize memory, storage, andprocessor resources of the server. The application may be defined usinga particular programming language and paradigm/model supported by theserver. However, building and deploying these types of applications isno longer desirable in most instances as many modern applications oftenneed to efficiently and securely scale (potentially across multipleplatforms) based on demand. There are many options for developingscalable, modern applications. Examples include, but are not limited to,virtual machines, microservices, and containers. The choice oftendepends on a variety of factors such as the type of workload, availableecosystem resources, need for automated scaling, compatible programminglanguage and paradigm/model, and/or execution preferences.

When developers select a containerized approach for creating scalableapplications, portions (e.g., microservices, larger services, etc.) ofthe application are packaged into containers. Each container maycomprise software code, binaries, system libraries, dependencies, systemtools, and/or any other components or settings needed to execute theapplication according to a particular model such as a declarative modelof programming. In this way, the container is a self-contained executionenclosure for executing that portion of the application.

Unlike virtual machines, containers do not include operating systemimages. Instead, containers ride on a host operating system which isoften light weight allowing for faster boot and utilization of lessmemory than a virtual machine. The containers can be individuallyreplicated and scaled to accommodate demand. Management of the container(e.g., scaling, deployment, upgrading, health monitoring, etc.) is oftenautomated by a container orchestration platform (e.g., Kubernetes).

The container orchestration platform can deploy containers on nodes(e.g., a virtual machine, physical hardware, etc.) that have allocatedcompute resources (e.g., processor, memory, etc.) for executingapplications hosted within containers. Applications (or processes)hosted within multiple containers may interact with one another andcooperate together. For example, a storage application within acontainer may access a deduplication application and an encryptionapplication within other containers in order deduplicate and/or encryptdata managed by the storage application. Container orchestrationplatforms often offer the ability to support these cooperatingapplications (or processes) as a grouping (e.g., in Kubernetes this isreferred to as a pod). This grouping (e.g., a pod) can supports multiplecontainers and forms a cohesive unit of service for the applications (orservices) hosted within the containers. Containers that are part of apod may be co-located and scheduled on a same node, such as the samephysical hardware or virtual machine. This allows the containers toshare resources and dependencies, communicate with one another, and/orcoordinate their lifecycles of how and when the containers areterminated.

DESCRIPTION OF THE DRAWINGS

Embodiments of the present technology will be described and explainedthrough the use of the accompanying drawings in which:

FIG. 1A is a block diagram illustrating an example of a distributedcontrol plane of control plane controllers paired with worker nodes inaccordance with an embodiment of the present technology.

FIG. 1B is a block diagram illustrating an example of a control planecontroller paired with a worker node in accordance with an embodiment ofthe present technology.

FIG. 1C is a block diagram illustrating an example of a control planecontroller and a cluster controller paired with a worker node inaccordance with an embodiment of the present technology.

FIG. 2 is a flow chart illustrating an example of a set of operationsthat route commands to control plane controllers paired with workernodes in accordance with various embodiments of the present technology.

FIG. 3 is a block diagram illustrating an example of a control planecontroller reformatting a command into a reformatted command that isthen routed to a worker node in accordance with an embodiment of thepresent technology.

FIG. 4 is an example of a custom resource definition in accordance withan embodiment of the present technology.

FIG. 5 is a flow chart illustrating an example of a set of operationsthat pair a new control plane controller with a new worker node inaccordance with various embodiments of the present technology.

FIG. 6 is a block diagram illustrating an example of pairing a newcontrol plane controller with a new worker node in accordance with anembodiment of the present technology.

FIG. 7 is a flow chart illustrating an example of a set of operationsthat remove a control plane controller and pod based upon a worker nodebeing removed from a distributed storage architecture in accordance withvarious embodiments of the present technology.

FIG. 8 is a flow chart illustrating an example of a set of operationsthat track ownership information of objects so that commands are routedto control plane controllers paired with worker nodes that are owners ofobjects targeted by the commands in accordance with various embodimentsof the present technology.

FIG. 9A is a block diagram illustrating an example of tracking ownershipinformation of objects so that commands are routed to control planecontrollers paired with worker nodes that are owners of objects targetedby the commands in accordance with an embodiment of the presenttechnology.

FIG. 9B is a block diagram illustrating an example of tracking ownershipinformation of objects so that commands are routed to control planecontrollers paired with worker nodes that are owners of objects targetedby the commands in accordance with an embodiment of the presenttechnology.

FIG. 10 is a flow chart illustrating an example of a set of operationsthat route commands to control plane controllers paired with workernodes based upon whether the worker nodes are operational or have failedin accordance with various embodiments of the present technology.

FIGS. 11A-11D are block diagrams illustrating an example of routingcommands to control plane controllers paired with worker nodes basedupon whether the worker nodes are operational or have failed inaccordance with an embodiment of the present technology.

FIG. 12 is a sequence diagram illustrating operation of a control planecontroller in accordance with an embodiment of the present technology.

FIG. 13 is a sequence diagram illustrating operation of a control planecontroller facilitating the creation a volume file in accordance with anembodiment of the present technology.

FIG. 14 is a sequence diagram illustrating operation of a control planecontroller facilitating the creation of a volume snapshot in accordancewith an embodiment of the present technology.

FIG. 15 is an example of a computer readable medium in which variousembodiments of the present technology may be implemented.

The drawings have not necessarily been drawn to scale. Similarly, somecomponents and/or operations may be separated into different blocks orcombined into a single block for the purposes of discussion of someembodiments of the present technology. Moreover, while the presenttechnology is amenable to various modifications and alternative forms,specific embodiments have been shown by way of example in the drawingsand are described in detail below. The intention, however, is not tolimit the present technology to the particular embodiments described. Onthe contrary, the present technology is intended to cover allmodifications, equivalents, and alternatives falling within the scope ofthe present technology as defined by the appended claims.

DETAILED DESCRIPTION

The techniques described herein are directed to implementing adistributed control plane to facilitate communication between acontainer orchestration platform and a distributed storage architecture.The demands on data center infrastructure and storage are changing asmore and more data centers are transforming into private and hybridclouds. Storage solution customers are looking for solutions that canprovide automated deployment and lifecycle management, scalingon-demand, higher levels of resiliency with increased scale, andautomatic failure detection and self-healing. To meet these objectives,a container-based distributed storage architecture can be leveraged tocreate a composable, service-based architecture that providesscalability, resiliency, and load balancing. The container-baseddistributed storage architecture may provide a scalable, resilient,software defined architecture that can be leveraged to be the data planefor existing as well as new web scale applications. The container-baseddistributed storage architecture may include a container orchestrationplatform (e.g., Kubernetes).

Applications may be deployed as containers within the containerorchestration platform in a scalable and on-demand manner. For example,a file system service application may be hosted within a container thatis managed by the container orchestration platform. The file systemservice application may be accessed by clients in order to store andretrieve data managed by the file system service application, such asthrough a volume. In order to provide these applications hosted by thecontainer orchestration platform with physical storage, a distributedstorage architecture is provided.

The distributed storage architecture may be hosted separate from andexternal to the container orchestration platform. This provides theability to tailor and configure the distributed storage architecture tomanage distribute storage in an efficient manner that can be madeaccessible to any type of computing environment, such as theapplications hosted within the container orchestration platform,applications and services hosted on servers or on-prem, applications andservices hosted within various types of cloud computing environments,etc. Accordingly, the distributed storage architecture is composed ofworker nodes that are configured to manage the distributed storage. Eachworker node may manage one or more storage devices, such as locallyattached storage. In this way, the storage devices of the worker nodesmay form the distributed storage. Data may be slice/distributed and/orredundantly stored across storage devices of multiple worker nodes,which may improve resilience to failures and/or enable more efficientload balancing. This is because a particular worker node may be assignedto be an owner of an object, such as a volume, stored across storagedevices of multiple worker node. If the work node fails, then ownershipof the object can be reassigned to another worker node for managing andproviding access to the object. Ownership of objects may be dynamicallychanged between worker nodes without physically migrating the data ofthe objects.

The distributed storage architecture implements and hosts the workernodes that manage the distributed storage, which may be used byapplications and services external to the distributed storagearchitecture for storing data. In some embodiments, volumes may becreated within the distributed storage of the distributed storagearchitecture. The applications hosted within the containers of thecontainer orchestration platform may mount to these volumes so that theapplications can store data within the volumes. Control plane logic canbe implemented to manage volume operations that are performed upon thevolumes stored by the worker nodes within the distributed storage of thedistributed storage architecture. These volume operations may correspondto volume creation operations, volume deletion operations, file creationoperations, volume snapshot creation operations, backup and restoreoperations, and/or other operations. The control plane logic acts as anintermediary layer that facilitates, tracks, and manages worker nodesexecuting control plane operations requested by the applications hostedwithin the containers in the container orchestration platform, such asthe creation of a volume within the distributed storage for use by anapplication.

Traditionally, the control plane logic may be hosted external to thecontainer orchestration environment, and thus is unable to leveragemanagement functionality, job scheduling services, APIs, resources,and/or other functionality and services provided by the containerorchestration environment for applications hosted within the containerorchestration environment. In order to incorporate and host the controlplane logic into the container orchestration environment, the controlplane logic could be hosted within a single control plane controllerhosted within a container of the container orchestration environment. Inthis way, various features provided by the container orchestrationenvironment for containers can be provided for the single control planecontroller, such as job scheduling, dynamic resource allocation/scaling,etc. Thus, the single control plane controller is a single centralizedcontroller for taking control and orchestrating all of the control planeoperations requested by the applications of the container orchestrationenvironment for execution by the worker nodes of the distributed storageenvironment. However, this solution of a single control plane controlleris unable to scale out to situations where each worker node may hostthousands of volumes, and there could be hundreds of worker nodes at anygiven moment. That is, a single control plane controller cannot scaleout to manage volume operations and other control plane operations forhundreds of worker nodes each hosting hundreds to thousands of volumes.Additionally, the single control plane controller would be a singlepoint of failure. If the single control plane controller or thecontainer hosting the single control plane controller crashes, then novolume level operations and/or other types of control plane operationshandled by the single control plane controller could be performed. Thus,if an application is attempting to create a volume for storing data,then the application would be unable to have the volume created, whichcould cause the application to error out or cause other problems.

Another issue with facilitating communication between the containerorchestration platform and the distributed storage architecture is thatthe container orchestration platform and the distributed storagearchitecture may utilize different models for defining and implementingprogramming commands. In some embodiments, the container orchestrationplatform (e.g., Kubernetes) may implement a declarative model (adeclarative programming model). With the declarative model, anapplication hosted within a container in the container orchestrationplatform can describe a result without explicitly listing instructions,programming commands, or executable steps to achieve the result. In someembodiments, an application may request the provisioning of a volume touse for storage. The request describes the volume, such as a name, size,and/or other attributes that the volume should have. However, therequest does not comprise the programming commands or executable stepsto actually create the volume. In contrast, the distributed storagearchitecture may implement an imperative model (an imperativeprogramming model). With the imperative model, a worker node expects andoperates based upon programming commands or executable steps (e.g.,statements that change a program's state) that are provided to theworker node to execute in order to accomplish a particular result. Insome embodiments, the worker node expects and is capable of executing aparticular sequence of programming commands or executable steps tocreate the volume. However, the worker node is unable perform to therequest, defined by the application accordingly to the declarativemodel, because these requests do not contain the required programmingcommands or executable steps that the worker node needs in order tocreate the volume.

Another issue with the control plane logic facilitating communicationbetween the container orchestration platform and the distributed storagearchitecture is the dynamic nature of the distributed storagearchitecture. The distributed storage architecture data of a volume maybe sliced/distributed across storage devices of multiple worker nodes.At any given point of there, there may be a single owner of the volume.Ownership of the volume can dynamically change amongst worker nodes suchas for load balancing or failover reasons. Traditional control planelogic does not understand this fluidity of volume ownership whereownership of a volume or other type of object can change even withoutmigrating data of the volume to the new owner. Thus, the traditionalcontrol plane logic is unable to handle volume ownership changes and/orfailover scenarios.

Accordingly, as provided herein, a distributed control plane isconfigured to facilitate communication between the containerorchestration platform and the distributed storage architecture in amanner that addresses the aforementioned issues and deficiencies oftraditional control plane logic. The distributed control plane is hostedwithin the container orchestration platform so that the distributedcontrol plane can leverage communication, job scheduling, dynamicresource allocation/scaling, containers, and/or other resources andservices provided by the container orchestration platform. At any givenpoint in time, the distributed control plane may comprise any number ofcontrol plane controllers that are hosted within pods of the containerorchestration platform (e.g., the number of controller plane controllersmay be dynamically scaled up or down based upon demand). In someembodiments, each control plane controller is paired with a singleworker node. This distributed aspect of the distributed control planewhere multiple control plane controllers may be utilized solves scalingand single point of failure issues that would otherwise arise if asingle control plane controller was used. Any number of control planecontrollers can be created and/or paired with worker nodes on-demand ina scale-out manner. Thus, if one of the control plane controllers fails,then a new control plane controller or an existing control planecontroller can take over for the failed control plane controller. Insome embodiments, any number of control plane controllers may be pairedwith any number of worker nodes.

The control plane controllers are configured with functionality that canreformat/convert commands formatted according to the declarative modelsupported by the container orchestration platform into reformattedcommands formatted according to the imperative model supported by thedistributed storage architecture, and vice versa. In some embodiments, avolume provisioning command may be created by an application within thecontainer orchestration platform by defining a custom resourcedefinition for a volume to be provisioned. The custom resourcedefinition is formatted according to the declarative model whereattributes of the volume are defined within the custom resourcedefinition, but the custom resource definition does not comprise theactual programming commands or executable steps that a worker node wouldneed to execute in order to actually provision the volume. Accordingly,a control plane controller is configured with functionality capable ofretrieving the attributes from the custom resource definition andutilizing those attributes to construct a reformatted command withprogramming commands or executable steps that the worker node canexecute to provision the volume with those attributes. This solvesissues where the container orchestration platform and the distributedstorage architecture utilize different programming models.

The distributed control plane is configured with functionality that cantrack the ownership of objects, such as volumes, by worker nodes. Thatis, an object may be owned by a single worker node at any given point intime. However, data of the object may be stored across storage devicesof multiple worker nodes. The distributed storage architecture maychange ownership of the object amongst worker nodes for various reasons,such as for load balancing or failover. When a command from anapplication targets a particular object, then the command is to berouted to the worker node owning that object. Configuring thedistributed control plane with the functionality that can track theownership of objects solves issues otherwise occurring when ownership ofan object changes to a different worker node, and thus commands(reformatted commands) targeting the object must be routed to thisdifferent worker node. In some embodiments, ownership informationmaintained by the distributed storage architecture queried using anidentifier of an object to determine that the identifier of the objectis paired with an identifier of a worker node.

The distributed control plane is configured with functionality that candetect worker node failures, addition of new worker nodes, and/orremoval of worker nodes. The ability to track when and how thedistributed storage architecture adds worker nodes, removes workernodes, or reacts to worker node failures allows the distributed controlplane to react accordingly. In some embodiments, if the distributedcontrol plane detects that that distributed storage architecturereplaces a failed worker node with a new worker node, then thedistributed control plane may reassign a control plane controller pairedwith the failed worker node to being paired with the new worker node ormay remove the control plane controller and create a new control planecontroller paired with the new worker node. In this way, the distributedcontrol plane can react to failures within the distributed storagearchitecture and/or dynamically scale up/down based upon the numberworker nodes currently operating within the distributed storagearchitecture.

Various embodiments of the present technology provide for a wide rangeof technical effects, advantages, and/or improvements to computingsystems and components. For example, various embodiments may include oneor more of the following technical effects, advantages, and/orimprovements: 1) a non-routine and unconventional distributed controlplane of control plane controllers that facilitate communication betweena container orchestration environment and a distributed storagearchitecture; 2) pairing the control plane controllers with worker nodesof the distributed storage architecture in a scalable manner with nosingle point of failure; 3) configuring the control plane controllerswith functionality that can reformat/convert commands formattedaccording to the declarative model supported by the containerorchestration platform into reformatted commands formatted according tothe imperative model supported by the distributed storage architecture,and vice versa; 4) configuring the distributed control plane withfunctionality that can track object ownership changes so that commandscan be dynamically routed to control plane controllers paired withworker nodes that are current owners of objects targeted by thecommands; and/or 5) configuring the distributed control plane withfunctionality that can detect worker node failures, addition of newworker nodes, and/or removal of worker nodes so that the distributedcontrol plane can react to failures within the distributed storagearchitecture and/or dynamically scale up/down based upon the numberworker nodes currently operating within the distributed storagearchitecture.

In the following description, for the purposes of explanation, newerspecific details are set forth in order to provide a thoroughunderstanding of embodiments of the present technology. It will beapparent, however, to one skilled in the art that embodiments of thepresent technology may be practiced without some of the specificdetails. While, for convenience, embodiments of the present technologyare described with reference to container orchestration platforms (e.g.,Kubernetes) and distributed storage architectures, embodiments of thepresent technology are equally applicable to various other types ofhardware, software, and/or storage environments.

The phrases “in some embodiments,” “according to some embodiments,” “inthe embodiments shown,” “in one embodiment,” and the like generally meanthe particular feature, structure or characteristic following the phraseis included in at least one implementation of the present technology,and may be included in more than one implementation period in additionsuch phrases do not necessarily refer to the same embodiment ordifferent embodiments.

FIG. 1A is a block diagram illustrating an example of a distributedcontrol plane of control plane controllers paired with worker nodes inaccordance with an embodiment of the present technology. A containerorchestration platform 102, such as Kubernetes, may be configured todeploy containers on nodes (e.g., a virtual machine, physical hardware,etc.) that have allocated compute resources (e.g., processor, memory,etc.) for executing applications hosted within the containers. In someembodiments, a first application 106 may be hosted within a firstcontainer 104, a second application 110 may be hosted within a secondcontainer 108, etc. The container orchestration platform 102 may supporta declarative model (a declarative programming model) of programmingcode. Accordingly, the applications hosted within the containers of thecontainer orchestration platform 102 may generate commands formattedaccording to the declarative model. In some embodiments, theapplications may generate commands to perform control plane operations,such as volume create and delete operations, file create operations,snapshot operations, backup and restore operations, or other volumeoperations according to the declarative model. In some embodiments, thefirst application 106 may generate a snapshot command to create asnapshot. The snapshot command may specify attributes of the snapshot,but does not include programming instructions or executable steps thatcan be executed to create the snapshot because the snapshot command isgenerated according to the declarative model.

The applications may utilize compute resources (e.g., processor, memory,etc.) provided by the container orchestration platform 102 to thecontainers hosting the applications. However, an application may needpersistent storage for storing data beyond what resources the containerorchestration platform 102 offers. Accordingly, a distributed storagearchitecture 112 is deployed for providing storage for the applicationshosted within the container orchestration platform 102. In someembodiments, the distributed storage architecture 112 is not hostedwithin the container orchestration platform 102 but may be hostedexternal to the container orchestration platform 102. The distributedstorage architecture 112 includes a plurality of work nodes, such as afirst worker node 114, a second worker node 116, a third worker node118, and/or other worker nodes. In some embodiments, the worker nodesmay be implemented as containers, virtual machines, serverless threads,or other hardware or software components. The worker nodes areconfigured to manage distributed storage 120 hosted by the distributedstorage architecture 112. The distributed storage 120 is comprised ofstorage of storage devices managed by (e.g., attached to) the workernodes, such as a first set of storage devices 122 of the first workernode 114, a second set of storage devices 124 of the second worker node116, a third set of storage devices 126 of the third worker node 118,etc. The distributed storage 120 may be made accessible to theapplications within the container orchestration platform 102. In someembodiments, a volume may be created within the distributed storage 120.The first application 106 may be provided with mountable access to thevolume so that the first application 106 can store and retrieve datafrom the volume. The data of the volume may be slice/distributed acrossstorage devices of one or more worker nodes. One of the worker nodes,such as the first worker node 114 may be designated as an owner of thevolume even though the data could be stored across storage devicesattached to other worker nodes.

The distributed storage architecture 112 may support an imperative model(an imperative programming model) of programming code. Thus, the workernodes of the distributed storage architecture 112 may be capable ofexecuting commands (formatted according to the imperative model, such ascommands comprising programming instructions or executable steps. Insome embodiments, a snapshot command formatted according to theimperative model may include programming instructions or executablesteps that a worker node can execute in order to create a snapshot.Because the distributed storage architecture 112 may not support thedeclarative model used by the applications of the containerorchestration platform 102, the worker nodes of the distributed storagearchitecture 112 may be unable to process commands defined by theapplications. As illustrated in the embodiments shown in FIG. 1A, adistributed control plane 128 is provided for reformatting commandsbetween the imperative model and the declarative model in order tofacilitate communication and execution of commands between theapplications and the worker nodes. The distributed control plane 128 mayinclude a plurality of control plane controllers that are configured toreformat/convert commands formatted according to the declarative modelsupported by the container orchestration platform 102 into reformattedcommands formatted according to the imperative model supported by thedistributed storage architecture 112, and vice versa.

In some embodiments of a control plane controller of the distributedcontrol plane 128 reformatting commands, the control plane controllermay receive a command formatted according to the declarative model. Thecontrol plane controller interprets the command to determine an intentof the command (e.g., a specified outcome, an objective of the command,a result that the command is to achieve, a purpose of the command,etc.), such as where the command has the intent for a volume object tobe provisioned. The intent may be identified based upon parameters,text, and/or other information within the command, such as where thecommand indicates that a volume object with a particular name and sizeis to be provisioned, but does not include instructions for how toprovision the volume object (e.g., an algorithm or text parsing functionmay be used to parse the command to identify a specified outcome of thecommand). The control plane controller compares the intent against acurrent state of the volume object, and issues reformatted commands tochange the current state of the volume object if necessary. For example,a reformatted command may change the current state to a provisioningstate to indicate that the volume object is currently being provisioned.The reformatted commands may comprise instructions that can be executedby a worker node of the distributed storage architecture 112 toprovision the volume object. The reformatted commands may be imperativecommands supported by the distributed storage architecture 112. In thisway, the control plane controller routes the imperative commands to theworker node of the distributed storage architecture 112 for execution.The imperative commands may be run as jobs by the worker node. Thecontrol plane controller may monitor the progress of the jobs, such asthe progress of long running jobs. If the control plane controllerdetects that a job has failed, then the control plane controller mayretry the job.

The distributed control plane 128 may be hosted within the containerorchestration platform 102. In some embodiment, each control planecontroller may be hosted within a pod of the container orchestrationplatform 102, such as where a first control plane controller 136 ishosted within a first pod 130, a second control plane controller 138 ishosted within a second pod 132, and a third control plane controller 140is hosted within a third pod 134. In this way, the control planecontrollers of the distributed control plane 128 are hosted within thecontainer orchestration platform 102 and may leverage resources,services, communication APIs, and/or other functionality of thecontainer orchestration platform 102. In some embodiments, each controlplane controller may be paired with a worker node according to a one toone pairing/relationship, such as where the first control planecontroller 136 is paired with the first worker node 114, the secondcontrol plane controller 138 is paired with the second worker node 116,and the third control plane controller 140 is paired with the thirdworker node 118. In some embodiments, there may be a one to many or manyto one pairing/relationship between control plane controllers and workernodes (e.g., a control plane controller paired with multiple workernodes or a worker node paired with multiple control plane controllers).Each worker node may be designated by the distributed storagearchitecture 112 as a current owner of certain objects such as volumesstored within the distributed storage 120. Accordingly, the distributedcontrol plane 128 is configured to route commands from applicationswithin the container orchestration platform 102 to control planecontroller paired with worker nodes that are current owners of objectstargeted by the commands. In this way, a control plane controller pairedwith a worker node can reformat commands targeting objects owned by theworker node and route the reformatted commands to the worker node toexecute.

FIG. 1B is a block diagram illustrating an example of a control planecontroller paired with worker node in accordance with an embodiment ofthe present technology. In some embodiments, the first control planecontroller 136 is paired with the first worker node 114 so that thefirst control plane controller 136 can communicate with the first workernode 114 through an API endpoint 150 of the first worker node 114. Insome embodiments, the API endpoint 150 may be a representational statetransfer (REST) API endpoint, and the first control plane controller 136transmits reformatted commands through REST API calls to the REST APIendpoint in order to communicate with the first worker node 114.

The first worker node 114 may comprise a data management system (DMS)152 and a storage management system (SMS) 158. The data managementsystem 152 is a client facing frontend with which clients (e.g.,applications within the container orchestration platform 102) interactthrough the distributed control plane 128, such as where reformattedcommands from the first control plane controller 136 are received at theAPI endpoint 150. The storage management system 158 is a distributedbackend (e.g., instances of the storage management system 158 may bedistributed amongst multiple worker nodes of the distributed storagearchitecture 112) used to store data on storage devices of thedistributed storage 120.

The data management system 152 may host one or more storage operatingsystem instances 154, such as a storage operating system instanceaccessible to the first application 106 through the first control planecontroller 136 for storing data. In some embodiments, the first storageoperating system instance may run on an operating system (e.g., Linux)as a process and may support various protocols, such as NFS, CIFS,and/or other file protocols through which clients may access filesthrough the storage operating system instance. The storage operatingsystem instance may provide an API layer through which applications mayset configurations (e.g., a snapshot policy, an export policy, etc.),settings (e.g., specifying a size or name for a volume), and transmitI/O operations directed to volumes 156 (e.g., FlexVols) exported to theapplications by the storage operating system instance. In this way, theapplications communicate through the control plane controller with thestorage operating system instance through this API layer. The datamanagement system 152 may be specific to the first worker node 114(e.g., as opposed to the storage management system (SMS) 158 that may bea distributed component amongst worker nodes of the distributed storagearchitecture). The storage operating system instance may comprise anoperating system stack that includes a protocol layer (e.g., a layerimplementing NFS, CIFS, etc.), a file system layer, a storage layer(e.g., a RAID layer), etc. The storage operating system instance mayprovide various techniques for communicating with storage, such asthrough ZAPI commands, REST API operations, etc. The storage operatingsystem instance may be configured to communicate with the storagemanagement system 158 through iSCSI, remote procedure calls (RPCs), etc.For example, the storage operating system instance may communicationwith virtual disks provided by the storage management system 158 to thedata management system 152, such as through iSCSI and/or RPC.

The storage management system 158 may be implemented by the first workernode 114 as a storage backend. The storage management system 158 may beimplemented as a distributed component with instances that are hosted oneach of the worker nodes of the distributed storage architecture 112.The storage management system 158 may host a control plane layer. Thecontrol plane layer may host a full operating system with a frontend anda backend storage system. The control plane layer may form a controlplane that includes control plane services, such as a slice service 162that manages slice files used as indirection layers for accessing dataon storage devices of the distributed storage 120, a block service 160that manages block storage of the data on the storage devices of thedistributed storage 120, a transport service used to transport commandsthrough a persistence abstraction layer to a storage manager 164, and/orother control plane services. The slice service 162 may be implementedas a metadata control plane and the block service 160 may be implementedas a data control plane. Because the storage management system 158 maybe implemented as a distributed component, the slice service 162 and theblock service 160 may communicate with one another on the first workernode 114 and/or may communicate (e.g., through remote procedure calls)with other instances of the slice service 162 and the block service 160hosted at other worker nodes within the distributed storage architecture112. Thus, the first worker node 114 may be a current owner of an object(a volume) whose data is sliced/distributed across storage device ofmultiple worker nodes, and the first worker node 114 can use the storagemanagement system 158 to access the data stored within the storagedevices of the other worker nodes by communicating with the otherinstances of the storage management system.

In some embodiments of the slice service 162, the slice service 162 mayutilize slices, such as slice files, as indirection layers. The firstworker node 114 may provide the applications, through the first controlplane controller 136, with access to a LUN or volume using the datamanagement system 152. The LUN may have N logical blocks that may be 1kb each. If one of the logical blocks is in use and storing data, thenthe logical block has a block identifier of a block storing the actualdata. A slice file for the LUN (or volume) has mappings that map logicalblock numbers of the LUN (or volume) to block identifiers of the blocksstoring the actual data. Each LUN or volume will have a slice file, sothere may be hundreds of slices files that may be distributed amongstthe worker nodes of the distributed storage architecture 112. A slicefile may be replicated so that there is a primary slice file and one ormore secondary slice files that are maintained as copies of the primaryslice file. When write operations and delete operations are executed,corresponding mappings that are affected by these operations are updatedwithin the primary slice file. The updates to the primary slice file arereplicated to the one or more secondary slice files. After, the write ordeletion operations are responded back to a client as successful. Also,read operations may be served from the primary slice since the primaryslice may be the authoritative source of logical block to blockidentifier mappings.

In some embodiments, the control plane layer may not directlycommunicate with the distributed storage 120, but may insteadcommunicate through the persistence abstraction layer to a storagemanager 164 that manages the distributed storage 120. In someembodiments, the storage manager 164 may comprise storage operatingsystem functionality running on an operating system (e.g., Linux). Thestorage operating system functionality of the storage manager 164 mayrun directly from internal APIs (e.g., as opposed to protocol access)received through the persistence abstraction layer. In some embodiments,the control plane layer may transmit I/O operations through thepersistence abstraction layer to the storage manager 164 using theinternal APIs. For example, the slice service 162 may transmit I/Ooperations through the persistence abstraction layer to a slice volumehosted by the storage manager 164 for the slice service 162. In thisway, slice files and/or metadata may be stored within the slice volumeexposed to the slice service 162 by the storage manager 164. In someembodiments, the storage management system 158 implements a masterservice 166 that performs cluster services amongst the worker nodes.

FIG. 1C is a block diagram illustrating an example of a control planecontroller and a cluster controller paired with worker node inaccordance with an embodiment of the present technology. A fourth workernode 170 of the distributed storage architecture 112 may be paired witha fourth control plane controller 172 hosted within a fourth pod 174 ofthe container orchestration platform 102. The fourth control planecontroller 172 may communicate with the fourth worker node 170 throughan API endpoint 184 such as a REST API endpoint. The fourth worker node170 may also be paired with a control server 176 hosting a clustermaster controller 178 within the container orchestration platform 102.The cluster master controller 178 may communicate with a cluster master180 of the fourth worker node 170 through an API endpoint 182 such as aREST API endpoint. The cluster master controller 178 may be configuredto handle certain types of operations, such as cluster creationcommands, add/remove worker node commands, add/remove storage commands,volume APIs for creating hierarchies of objects being created by thevolume APIs, and/or cluster management commands. Thus, the fourthcontrol plane controller 172 may handle certain types of operations,while the cluster master controller 178 may handle other types ofoperations

In some embodiments, a worker node may be designated as a clustermaster, such as the fourth worker node 170 hosting the cluster master180. The cluster master controller 178 and the cluster master 180 may beconfigured to implement commands corresponding to infrastructure APIs,such as the cluster creation commands, the add/remove worker nodecommands, the add/remove storage commands, the volume APIs for creatinghierarchies of objects being created by the volume APIs, and/or clusterthe management commands. In some embodiments, the cluster mastercontroller 178 and the cluster master 180 may perform certain operationsassociated with commands corresponding to volume APIs. The clustermaster controller 178 and the cluster master 180 may create a hierarchyof objects for volumes created by the volume APIs and may preservevolume core identifiers of the volumes across the plurality of workernodes and/or control plane controllers. In this way, the volume coreidentifiers can be used by any worker node to identify the volumes.

FIG. 2 is a flow chart illustrating an example of a set of operationsthat route commands to control plane controllers paired with workernodes in accordance with various embodiments of the present technology.The worker nodes of the distributed storage architecture 112 and theapplications within the container orchestration platform 102 may utilizedifferent programming models, and thus commands from the applicationscannot natively be processed by the worker nodes. To solve this problem,the control plane controllers of the distributed control plane 128 areconfigured to reformat/translate the commands so that the commands canbe interpreted and executed by the worker nodes. The distributed controlplane 128 may be hosted within the container orchestration platform 102that is also hosting the applications. In some embodiments, the controlplane controllers of the distributed control plane 128 may beimplemented as plug-ins to the container orchestration platform 102. Aplug-in used to implement a control plane controller may be providedwith access to a worker node through a REST API endpoint, such as theAPI endpoint 150 of FIG. 1B.

During operation 202 of method 200, the distributed control plane 128may receive a command from an application hosted within a container ofthe container orchestration platform 102. In some embodiments, thecommand may correspond to a control plane operation, such as a commandto provision a volume, a file command targeting a file, a snapshotcreate command to create a snapshot of a volume, a command to create orapply an export policy for a volume to control client access to thevolume, a command to create a backup, a command to perform a restoreoperation, a cluster creation, deletion, or modification command, acommand to add or remove storage, a command to add or remove a workernode, etc. The command may be formatted according to the declarativemodel supported by the container orchestration platform 102 and used bythe applications to generate commands.

In some embodiments of receiving the command from the application, theapplication may generate the command, which is routed through thecontainer orchestration platform 102 to the distributed control plane128. In some embodiments of receiving the command from the application,a custom resource definition maintained within a distributed databasehosted within the container orchestration platform 102 may be created ormodified in order to define the command through the custom resourcedefinition (as opposed to generating and transmitting the command). Forexample, the application may create a new custom resource definition forprovisioning a volume within the distributed storage 120 of thedistributed storage architecture 112 for use by the application. The newcustom resource definition may be defined according to the declarativemodel such as through a custom resource specification listing attributesof the volume to create (e.g., volume name, volume size, etc.).

In some embodiments, custom resource definitions may correspond to acluster custom resource definition, a volume custom resource definition,an export policy custom resource definition, a snapshot custom resourcedefinition, a cloud backup custom resource definition, or otherdefinitions of custom resources (e.g., a storage structure, datastructure, functionality, or resource not natively supported by thestorage orchestration platform). The distributed control plane 128 maymonitor the distributed database for changes, such as the creation ofthe new custom resource definition or modifications to existing customresource definitions. Upon detecting the new custom resource definition(or a modification to an existing custom resource definition), thedistributed control plane 128 may extract information such as attributesfrom fields within a custom resource specification of the new customresource definition as the control plane operation. The information mayrelate to volume information of the volume, cluster information forhosting the volume, volume name information, export policy informationto manage access to the volume, permissions information for accessingthe volume, quality of service policy information for hosting thevolume, volume size information of the volume, or other information usedto define a control plane operation. In this way, the control planeoperation derived from the information extracted from the customresource specification of the custom resource definition is received bythe distributed control plane 128 as the command.

Once the distributed control plane 128 has received the command, thedistributed control plane 128 may determine, during operation 204 ofmethod 200, whether the command targets an object owned by the firstworker node 114 or the second worker node 116 (or a different workernode). In some embodiments, the distributed control plane 128 evaluatesobject ownership information to identify which worker node is an ownerof an object targeted by the command (e.g., an owner of a volume beingmodified, an owner of a file being operated upon, an owner of a volumebeing snapshotted, an owner of a backup being used to perform a restore,a worker node that is to host a volume being created, etc.). The objectownership information may be maintained by the distribute storagearchitecture 112, which may be evaluated for each command received inorder to identify an owner of the object targeted by the command asownership can change over time. In this way, the distributed controlplane 128 can identify a control plane controller paired with the workernode currently owning the object targeted by the command.

If the first worker node 114 is the owner of the object targeted by thecommand, then the distributed control plane 128 may route the command tothe first control plane controller 136, during operation 206 of method200. During operation 208 of method 200, the first control planecontroller 136 reformats the command from being formatted according tothe declarative model to being formatted according to the imperativemodel as a reformatted command. In some embodiments, the informationextracted from a custom resource definition as the control planeoperation of the command may be used to construct executable operations,functions, and/or other imperative programming steps that can beexecuted by the first worker node 114 to perform/execute the reformattedcommand. During operation 210 of method 200, the first control planecontroller 136 transmits the reformatted command, such as through a RESTAPI call, to the API endpoint 150 of the first worker node 114 for thefirst worker node 114 to implement the control plane operation definedwithin the reformatted command according to the imperative model. Insome embodiments, the REST API call includes a security certificateand/or credentials used to authenticate with the first worker node 114.In some embodiments, the first control plane controller 136 may createand monitor a job that the first worker node 114 performs in order toimplement the control plane operation based upon the reformattedcommand. In this way, the first control plane controller 136 can trackthe status of performing the reformatted command by monitoring the job.

If the second worker node 116 is the owner of the object targeted by thecommand, then then the distributed control plane 128 may route thecommand to the second control plane controller 138 (or a control planecontroller paired with the current owner worker node), during operation212 of method 200. During operation 214 of method 200, the secondcontrol plane controller 138 reformats the command from being formattedaccording to the declarative model to being formatted according to theimperative model as a reformatted command. In some embodiments, theinformation extracted from a custom resource definition as the controlplane operation of the command may be used to construct executableoperations, functions, and/or other imperative programming steps thatcan be executed by the second worker node 116 to perform/execute thereformatted command. During operation 216 of method 200, the secondcontrol plane controller 138 transmits the reformatted command, such asthrough a REST API call, to an API endpoint of the second worker node116 for the second worker node 116 to implement the control planeoperation. In some embodiments, the REST API call includes a securitycertificate and/or credentials used to authenticate with the secondworker node 116. In some embodiments, the second control planecontroller 138 may create and monitor a job that the second worker node116 performs in order to implement the control plane operation basedupon the reformatted command. In this way, the second control planecontroller 138 can track the status of performing the reformattedcommand by monitoring the job.

In some embodiments, a control plane controller that has transmitted areformatted command to a worker node for implementation of a controlplane operation may receive a response from the worker node. Theresponse may comprise information relating to a current status (progresscompletion) of implementing the control plane operation, a result ofcompleting the implementation of the control plane operation, warninginformation relating to implementing the control plane operation (e.g.,a volume that is to be provisioned consumes more space than allowed, thevolume has the same name as an existing volume, a snapshot targets avolume that does not exist, an export policy is being applied to avolume that does not exist, etc.), state information of the object(e.g., attributes of a volume that has been provisioned or a snapshotthat has been created), etc. The control plane controller may conveythis information back to the application requesting performance of thecommand by populating the information into a custom resource definitionfor the object (e.g., the volume, the snapshot, the export policy, etc.)targeted by the command. In some embodiments, the warning information orstate information of the object may be populated within an event fieldof the custom resource definition. In some embodiments, otherinformation may be populated within a status field of the customresource definition, such as a create time, a name, an export policy, anexport address, permission information, a quality-of-service policy, asize, a state of a volume, a path of a volume, etc. In this way, thecontrol plane controller is used as an intermediary device forreformatting communicate between the application and the worker node,such as for facilitating the performance of commands to create newobjects (create a volume), modify existing volumes, creating snapshots,creating clones, creating or applying export polices, etc. As will bedescribed in further detail, FIG. 4 illustrates an example of a customresource definition.

FIG. 3 is a block diagram illustrating an example of a control planecontroller reformatting a command into a reformatted command routed to aworker node in accordance with an embodiment of the present technology.The first application 106 may generate a command 302 to create or modifyan object. In some embodiments, the first application 106 may define thecommand 302 by creating a new custom resource definition or modifying anexisting custom resource definition for the object. The custom resourcedefinition may be stored within a distributed database 304 within thecontainer orchestration platform 102. The distributed control plane 128may receive the command 302, such as by extracting information from thecustom resource definition to derive a control plane operation of thecommand 302. The distributed control plane 128 may evaluate ownershipinformation of objects to identify the second worker node 116 as acurrent owner of the object targeted by the command. Accordingly, thedistributed control plane 128 may route the command 302 to the secondcontrol plane controller 138 paired with the second worker node 116. Thesecond control plane controller 138 may reformat the command 302 as areformatted command 306 that is transmitted to the second worker node116 to implement the control plane operation.

FIG. 4 is an example of a custom resource definition in accordance withan embodiment of the present technology. A custom resource definition402 may be used to define custom objects (custom resources) within thecontainer orchestration platform 102 (Kubernetes), such as to define avolume custom object. The custom object provides the ability to extendnative capabilities (beyond standard objects natively supported byKubernetes) of the container orchestration platform 102 (Kubernetes) bycreating and adding any type of API object as a custom object. Forexample, Kubernetes may natively provide a Kubernetes volume as adirectory or block device mounted inside a container running in a pod.This Kubernetes volume is a native Kubernetes object and is not a customobject defined through a custom resource definition. Kubernetes volumesrepresent physical devices managed by Kubernetes.

Various embodiments can use a custom resource definition to extendnative Kubernetes capabilities in order to define and create a volume asa custom object that can be used by an application. This volume may bereferred to as a volume custom object that is not a native Kubernetesobject. This provides the ability to extend Kubernetes capabilitiesbeyond the default native Kubernetes capabilities and standard objectsnatively supported by Kubernetes. In some embodiments, the customresource definition may be created through a .yaml file, and comprisesvarious fields used to define the volume custom object. Various types ofcustom objects may be defined through custom resource definitions, suchas volumes, snapshots, nodes, clusters, backup functionality, restorefunctionality, etc. These custom objects (custom resources) defined bythe custom resource definitions may be stored within the distributedstorage 120 of the distributed storage architecture 112.

The custom resource definition 402 may comprise a custom resourcespecification 404 for a volume (e.g., a volume clone), which may bepopulated with information such as a volume clone identifier, a clustername, a display name, an export policy, permissions information, aquality of service policy, a size of the volume, a snapshot reservepercentage (e.g., an amount of storage reserved for snapshots of thevolume), access types allowed for the volume, a volume path of thevolume, etc. In some embodiments, the custom resource specification 404may be populated by an application to define a command (a control planeoperation) targeting the volume or to define/provision the volume. Thecustom resource definition 402 may comprise a status field 406 for thevolume (e.g., the volume clone), which may be populated with informationsuch as the volume clone identifier, the cluster name, conditions (e.g.,a last transition time, a message of whether the volume is online, areason for the message such as because the volume is online, a status ofthe message such as the message being true, a type of the message suchas a volume_online_type, etc.), whether the volume was successfullycreated, a display name, an export address, an export policy, aninternal name, permissions information, a quality of service policy, arequested volume size, a restore cache size, a size of the volume, asnapshot reserve percentage, a state of the volume, a volume path, avolume UUID, etc. The status field 406 may be populated by a controlplane controller with information from a response received by a workernode that implemented a control plane operation to provision the volume.In this way, the status field 406 may be used by the control planecontroller to communicate information to the application regardingexecution of the control plane operation. Similarly, the control planecontroller can populate an events field 408 with state information ofthe volume and/or warning information relating the execution of thecontrol plane operation (e.g., a size of the volume being provisioned istoo large, a name for the volume is already assigned to an existingvolume, etc.).

FIG. 5 is a flow chart illustrating an example of a set of operationsthat pair a new control plane controller with a new worker node inaccordance with various embodiments of the present technology. Duringoperation 502 of method 500, control plane controllers, hosted withinthe container orchestration platform 102, may be paired with workernodes of the distributed storage architecture 112 according to a one orone relationship where a single control plane controller is paired witha single worker node. It may be appreciated that other pairingrelationships are contemplated, such as where multiple control planecontrollers are paired with a single worker node, or a control planecontroller is paired with multiple worker nodes. In some embodiments ofinitially constructing the distributed control plane 128 with controlplane controllers and/or subsequently modifying the distributed controlplane 128, the distributed storage architecture 112 may be evaluated toidentify a number of worker nodes hosted by the distributed storagearchitecture 112. For each worker node not already paired with a controlplane controller, a pod (a container managed by a pod) may be createdwithin the container orchestration platform 102. A control planecontroller may be hosted within the pod (a newly created pod or anexisting pod with additional resources to host the control planecontroller). The control plane controller may be paired with a workernode not already paired with a control plane controller. In this way,the control plane controller is configured to communicate and formatcommands between the worker node and applications within the containerorchestration platform 102 based upon the commands targeting objectscurrently owned by the worker node. Because worker nodes can bedynamically added and removed from the distributed storage architecture112, the distributed control plane 128 may be configured to create orremove control plane controllers in order to scale up or down based upona current number of worker nodes of the distributed storage architecture112.

The distributed control plane 128 is configured to selectively routecommands to control plane controllers that are paired with worker nodesthat are current owners of object targeted by the commands. In someembodiments, if a command relates to a volume provisioning task tocreate a volume, then the distributed control plane 128 routes thecommand to a worker node designated to be an owner of the volume. Insome embodiments, the distributed control plane 128 may track ownershipof objects that are owned by particular worker nodes. This ownershipinformation can be used to identify which worker node owns an object(e.g., owner of a volume, a file, a snapshot or backup that can be usedto perform a restore operation, a worker node to host a new volume, aworker node to create and manage a snapshot, etc.) so that a commandtargeting the object can be routed to a control plane controller pairedwith that worker node. Ownership of objects can dynamically changeamongst worker nodes, and thus the distributed control plane 128 mayupdate the ownership information over time to reflect such ownershipchanges.

During operation 504 of method 500, the distributed control plane 128may determine whether a new worker node has been added to thedistributed storage architecture 112. If the new worker node has beenadded to the distributed storage architecture 112, then the distributedcontrol plane 128 may create a new control plane controller configuredto reformat commands to create reformatted commands formatted accordingto the imperative model of programming, during operation 506 of method500. In some embodiments, a new pod may be created to host the newcontrol plane controller within the container orchestration platform 102or the new control plane controller may be hosted within an existing podor container with resources available for hosting the new control planecontroller. Compute and/or other resources of the containerorchestration platform 102 may be assigned to the pod for use by the newcontrol plane controller. During operation 508 of method 500, the newcontrol plane controller may be paired with the new worker node so thatthe new control plane controller can communicate with the new workernode through an API endpoint (a REST API endpoint) of the new workernode.

During operation 510 of method 500, the new control plane controller maybe configured to create and route reformatted commands to the new workernode based upon the new worker node owning objects targeted by thereformatted commands. In some embodiments, the new control planecontroller may generate a reformatted command from a command to performa control plane operation. The control plane operation may target avolume owned by the new worker node. Data of the volume may besliced/distributed across storage of multiple worker nodes of thedistributed storage architecture 112. In some embodiments, the controlplane operation may be executed by the new worker node to create asnapshot of the volume whose data is sliced across the storage devicesof the multiple worker nodes. The new control plane controller may beconfigured to populate a custom resource definition maintained withinthe container orchestration platform 102 for the object targeted by thecontrol plane operation. The custom resource definition may be populatedwith information received within a response from the new worker nodeexecuting the control plane operation, such as status information, eventinformation such as warning or state information, etc.

If the distributed control plane 128 determines that no new worker nodeshave been added to the distributed storage architecture 112, thencommands may be routed to control plane controllers paired with existingworker nodes owning objects targeted by the commands, during operation512 of method 500. During operation 514 of method 500, the control planecontrollers reformat commands into reformatted commands according to theimperative model supported by the distributed storage architecture 112.In some embodiments, the control plane controllers may reformat commandscorresponding to setting polices for objects, such as a security policy,an export policy, or access control list used to determine whatapplications are allowed to access certain objects. During operation 516of method 500, the control plane controllers route the reformattedcommands to the worker nodes paired with the control plane controllers.In some embodiments, commands may be processed by control planecontrollers in parallel based upon the commands targeting objects ownedby different worker nodes to which the control plane controllers arepaired, which improves throughput and processing of commands due toparallel processing of the commands.

FIG. 6 is a block diagram illustrating an example of pairing a newcontrol plane controller with a new worker node in accordance with anembodiment of the present technology. The distributed control plane 128may be configured to monitor the distributed storage architecture 112 todetermine whether the distributed storage architecture 112 has added orremoved worker nodes. In some embodiments, the distributed control plane128 may determine that the distributed storage architecture 112 added anew worker node 604 and new storage devices 606 for the new worker node604. Accordingly, the distributed control plane 128 may create a new pod608 (or new container) and may host a new control plane controller 610within the new pod 608. The new control plane controller 610 may beconfigured to facilitate communication of commands between theapplications within the container orchestration platform 102 and the newworker node 604 based upon the commands targeting objects owned or to beowned by the new worker node 604. Such object ownership information 602may be tracked by the distributed control plane 128.

FIG. 7 is a flow chart illustrating an example of a set of operationsthat remove a control plane controller and pod based upon a worker nodebeing removed from a distributed storage architecture in accordance withvarious embodiments of the present technology. During operation 702 ofmethod 700, control plane controllers, hosted within the containerorchestration platform 102, may be paired with worker nodes of thedistributed storage architecture 112 according to a one or onerelationship where a single control plane controller is paired with asingle worker node. During operation 704 of method 700, the distributedcontrol plane 128 may determine whether a worker node has been removedfrom the distributed storage architecture 112. If the worker node hasbeen removed from the distributed storage architecture 112, then acontrol plane controller paired with the worker node may be removed fromthe container orchestration platform 102, during operation 706 of method700. If a pod hosting the control plane controller is not hosting othercontrol plane controllers, then the pod may be removed from thecontainer orchestration platform 102, during operation 708 of method700.

If the distributed control plane 128 determines that no worker nodeshave been removed from the distributed storage architecture 112, thenthe control plane controller may route commands to control planecontrollers paired with worker nodes owning objects targeted by thecommands, during operation 710 of method 700. The control planecontrollers may reformat the commands according to an imperative modelsupported by the distributed control plane 128, during operation 712 ofmethod 700. The control plane controllers may transmit the reformattedcommands to API endpoints of the worker nodes paired with the controlplane controllers, during operation 714 of method 700.

FIG. 8 is a flow chart illustrating an example of a set of operationsthat track ownership information of objects so that commands are routedto control plane controllers paired with worker nodes that are owners ofobjects targeted by the commands in accordance with various embodimentsof the present technology. The distributed control plane 128 mayidentify and/or track ownership information of objects stored within thedistributed storage 120. The ownership information may indicate whichworker nodes within the distributed storage architecture 112 are currentowners of objects, such as volumes, snapshots, export policies, and/orother types of objects maintained on behalf of the applications hostedwithin the container orchestration platform 102. Data of an object maybe stored within storage devices of one or more worker nodes, but asingle worker node may be designated as an owner of the object at anygiven point in time. Ownership of the object may change for variousreason, such as due to load balancing by the distributed storagearchitecture 112 or due the distributed storage architecture 112 failingover ownership of objects from a failed worker node to a differentworker node. In some embodiments, ownership of an object may change fromone worker node to another worker node while retaining data of theobject within current storage locations within the distributed storage120. Thus, object ownership may change without having to migrate data ofthe object.

In some embodiments, the distributed control plane 128 may identify andtrack the ownership information by requesting object owner informationfrom the distributed storage architecture 112 or the containerorchestration platform 102. In some embodiments, the distributed controlplane 128 may periodically poll the distributed storage architecture 112or the container orchestration platform 102 to obtain updated objectownership information. In some embodiments, the distributed controlplane 128 may execute a polling thread at a cluster level to detectobject ownership changes of objects amongst worker nodes that form acluster within the distributed storage architecture 112. In someembodiments, if the distributed control plane 128 determines thatownership of an object has changed from the first worker node 114 to thethird worker node 118, then the distributed control plane 128 may updatethe ownership information such as by modifying an entry within theownership information for the object to now map to an identifier or nameof the third worker node 118. Each object within the ownershipinformation may have an entry mapping an identifier or name of an objectto an identifier or name of a worker node currently owning the object.In some embodiments, the distributed control plane 128 may detect afailure of the second worker node 116 (e.g., a loss of a heartbeat,receiving a failure notification from the distributed storagearchitecture 112, a certain amount of time lapsing without receiving aresponse from the second worker node 116, etc.). Accordingly, thedistributed control plane 128 may identify one or more worker nodes thatare now the new owners of objects that were owned by the second workernode 116 and may update the ownership information accordingly.

Ownership of objects may be identified and/or determined periodicallyand/or dynamically when a command has been received from an applicationby the distributed control plane 128. During operation 802 of method800, the distributed control plane 128 may receive a command from anapplication. The distributed control plane 128 may evaluate the commandto identify an object as a target of the command. In some embodiments,the command may comprise a volume identifier or name of a volumetargeted by the command. In some embodiments, the command may be asnapshot create command to create a snapshot of the volume. In someembodiments, the command may be a file operation command targeting afile stored within the distributed storage 120. In some embodiments, thecommand is a provision command to create a volume that is exposed to theapplication for mountable access as a network file system (NFS) volume.In some embodiments, the command may target a volume whose data issliced/distributed across storage devices of multiple worker nodes, anda worker node that is designated as an owner of the volume may be one ofthe worker nodes or a different worker node whose storage devices do notstore data of the volume.

During operation 804 of method 800, the distributed control plane 128may determine whether the ownership information indicates that ownershipof the object targeted by the command has changed. If ownership of theobject has not changed, then the command is routed to a control planecontroller paired with the worker node owning the object, duringoperation 806 of method 800. During operation 808 of method 800, thecontrol plane controller may reformat the command according to theimperative model supported by the distributed storage architecture 112to create a reformatted command. During operation 810 of method 800, thecontrol plane controller may transmit the reformatted command to an APIendpoint of the worker node for implementation. In some embodiments, theworker node may execute a control plane operation defined by thereformatted command as executable instructions or programming steps thatthe worker node can execute according to the imperative model upon theobject, such as to create a snapshot of a volume whose data issliced/distributed across storage devices of multiple worker nodes.

If ownership of the object has changed, then the distributed controlplane 128 identifies the current worker node now owning the object,during operation 812 of method 800. During operation 814 of method 800,the distributed control plane 128 routes the command to a control planecontroller paired with the current worker node now owning the object.During operation 816 of method 800, the control plane controller mayreformat the command according to the imperative model supported by thedistributed storage architecture 112 to create a reformatted command.During operation 818 of method 800, the control plane controller maytransmit the reformatted command to an API endpoint of the currentworker node for implementation.

FIG. 9A is a block diagram illustrating an example of tracking ownershipinformation of objects so that commands are routed to control planecontrollers paired with worker nodes that are owners of objects targetedby the commands in accordance with an embodiment of the presenttechnology. The distributed control plane 128 may be provided withaccess to ownership information 602 of objects stored within thedistributed storage 120 of the distributed storage architecture 112.When the distributed control plane 128 receives a command 902 from anapplication hosted within the container orchestration platform 102, thedistributed control plane 128 may evaluate the ownership information 602to identify a worker node that is a current owner of an object targetedby the command 902. In some embodiments, the distributed control plane128 may executed a polling thread 901 to identify the ownershipinformation 602 for identifying the worker node that is the currentowner of the object targeted by the command 902. In some embodiments,the ownership information 602 may indicate that the second worker node116 is the current owner 906 of the object. Accordingly, the distributedcontrol plane 128 may route the command 902 to the second control planecontroller 138 paired with the second worker node 116. The secondcontrol plane controller 138 may reformat the command 902 to create areformatted command 904 that is transmitted to the second worker node116 for execution.

FIG. 9B is a block diagram illustrating an example of tracking ownershipinformation of objects so that commands are routed to control planecontrollers paired with worker nodes that are owners of objects targetedby the commands in accordance with an embodiment of the presenttechnology. The distributed control plane 128 may receive a subsequentcommand 910 that is received after the command 902. The distributedcontrol plane 128 may determine that the subsequent command 910 targetsthe same object that the command 902 targeted. The distributed controlplane 128 may evaluate the ownership information 602 to identify acurrent owner of the object. The ownership information 602 may indicatethat ownership of the object changed from the second worker node 116 tothe first worker node 114. Accordingly, the distributed control plane128 may route the subsequent command 910 to the first control planecontroller 136 paired with the first worker node 114 that is the currentowner 914 of the object. The first control plane controller 136 mayreformat the subsequent command 910 to create a reformatted subsequentcommand 912 that is transmitted to the first worker node 114 forexecution.

FIG. 10 is a flow chart illustrating an example of a set of operationsthat route commands to control plane controllers paired with workernodes based upon whether the worker nodes are operational or have failedin accordance with various embodiments of the present technology. Thedistributed control plane 128 may be configured to identify and/or trackthe status of worker nodes of the distributed storage architecture 112,such as health status information of the worker nodes. The health statusinformation may indicate whether a worker node is operational, failed,overloaded and operating with degraded performance, etc. In someembodiments, the distributed control plane 128 tracks the health statusinformation by polling the distributed storage architecture 112 forstatus information of the worker nodes. In some embodiments, the clustermaster 180 of the fourth worker node 170, the master service of otherworker nodes such as the master service 166 of the first worker node114, and/or the cluster master controller 178 of the control server 176may track the status of the worker nodes, which may be used as thehealth status information of the worker nodes. These components mayexchange operation status information, utilize heartbeats to determinewhether a worker node is operational, and/or utilize various othermechanisms to determine the statuses (operational states) of the workernodes. In some embodiments, the distributed control plane 128 mayutilize the polling thread 901 to identify the statuses of the workernodes and/or identify object ownership changes of objects amongst thework nodes such as due to worker nodes failing.

In some embodiments, the distributed storage architecture 112 maytracking the health status information of the worker nodes, and providethe health status information to the distributed control plane 128. Insome embodiments, the distributed storage architecture 112 may utilizethe health status information to detect a failure of a worker node (oran underperforming worker node). The distributed storage architecture112 may transfer ownership of at least some objects owned by the workernode if the worker node is underperforming in order to reduce load onthe worker node. The distributed storage architecture 112 may transferownership of all objects owned by the worker node to other worker nodes(to existing worker nodes or newly created worker nodes to replace thefailed worker node) based upon the worker node failing. The distributedstorage architecture 112 may replace the worker node with one or morenew worker nodes based upon the work node failing. The distributedstorage architecture 112 may remove the failed worker node.

Information relating to a worker node failing, a worker node beingreplaced by a new worker node, a worker node being removed, and/orownership of objects being transferred from a failed worker node to anew or existing worker node may be provided to the distributed controlplane 128. The distributed control plane 128 may implement variousactions based upon this information. In some embodiments, thedistributed control plane 128 may determine that the distributed storagearchitecture 112 removed a worker node such as due to the worker nodefailing. Accordingly, the distributed control plane 128 may remove acontrol plane controller that was paired with the worker node and/or mayremove a pod hosting the control plane controller if the pod is nolonger being used. In some embodiments, the distributed control plane128 may determine that the distributed storage architecture 112 added anew worker node such as to replace a failed worker node. The distributedcontrol plane 128 may remove a control plane controller that was pairedwith the failed worker node and/or may remove a pod hosting the controlplane controller if the pod is no longer being used. The distributedcontrol plane 128 may create and pair a new control plane controllerwith the new worker node. Or instead of removing the control planecontroller, the distributed control plane 128 may change (reassign) thecontrol plane controller from being paired with the failed worker nodeto being paired with the new worker node.

The distributed control plane 128 may receive a command to perform acontrol plane operation associated with an object stored within thedistributed storage architecture 112, during operation 1002 of method1000. The distributed control plane 128 may identify a worker node as anowner of the object targeted by the command. During operation 1004 ofmethod 1000, the distributed control plane 128 may determine whether theworker node owning the object targeted by the command has failed. If theworker node has not failed, then the command is routed to a controlplane controller paired with the worker node owning the object, duringoperation 1006 of method 1000. During operation 1008 of method 1000, thecontrol plane controller may reformat the command according to animperative model supported by the distributed storage architecture 112as a reformatted command. During operation 1010 of method 1000, thecontrol plane controller may transmit the reformatted command to an APIendpoint of the worker node for performing the control plane operation.

If the worker node has failed, then ownership of the object istransferred to a replacement worker node, during operation 1012 ofmethod 1000. The object may be transferred to an existing worker node ora newly created worker node to replace the failed worker node as thereplacement worker node. If the replacement worker node is not currentlypaired with a control plane controller such as because the replacementworker node is a new worker node, then a control plane controller ispaired with the replacement worker node, during operation 1014 of method1000. If the replacement worker node is already paired with a controlplane controller, then the distributed control plane 128 identifies thecontrol plane controller paired with the replacement worker node. Duringoperation 1016 of method 1000, the command is routed to the controlplane controller paired with the replacement worker node. Duringoperation 1018, the control plane controller reformats and transmits thecommand as a reformatted command to an API endpoint of the replacementworker node for performing the control plane operation.

FIG. 11A is a block diagram illustrating an example of routing commandsto control plane controllers paired with worker nodes based upon whetherthe worker nodes are operational or have failed in accordance with anembodiment of the present technology. The distributed control plane 128may receive a command 1102 from an application hosted within thecontainer orchestration platform 102. In some embodiments, the command1102 may be formatted according to a declarative model as attributesspecified through a custom resource specification of a custom resourcedefinition within the distributed database 304 of the containerorchestration platform 102. The distributed control plane 128 mayutilize a polling thread 901 to determine ownership information 602 ofan object targeted by the command 1102. The ownership information 602may indicate that the second worker node 116 is a current owner of theobject targeted by the command 1102. The distributed control plane 128may evaluate health status information of the second worker node 116 todetermine whether the second worker node 116 is operational (availableto perform the command 1102) or has failed. In response to determiningthat the second worker node 116 is operational, the distributed controlplane 128 may route the command 1102 to the second control planecontroller 138 paired with the second worker node 116. The secondcontrol plane controller 138 may reformat the command 1102 to create areformatted command 1104 formatted according to the imperative modelsupported by the second worker node 116. The second control planecontroller 138 transmits the reformatted command 1104 to the secondworker node 116 to execute.

FIG. 11B is a block diagram illustrating an example of routing commandsto control plane controllers paired with worker nodes based upon whetherthe worker nodes are operational or have failed in accordance with anembodiment of the present technology. The distributed control plane 128may determine that the second worker node 116 has failed 1110. In someembodiments, the distributed control plane 128 may receive anotification from the distributed storage architecture 112 that thesecond worker node 116 failed 1110. In some embodiments, the distributedcontrol plane 128 may detect the failure 1110 such as through a loss ofa heartbeat shared by the second worker node 116. In some embodiments,the distributed control plane 128 may evaluate health status informationto determine that the second worker node 116 failed 1110.

FIG. 11C is a block diagram illustrating an example of routing commandsto control plane controllers paired with worker nodes based upon whetherthe worker nodes are operational or have failed in accordance with anembodiment of the present technology. The distributed storagearchitecture 112 may reassign ownership of the object from the failedsecond worker node 116 to the first worker node 114 as a new objectowner 1120 of the object. The reassignment of the object to the firstowner node 114 as the new object owner 1120 of the object may bereflected within the ownership information 602 such as where an entryfor the object is updated with an identifier or name of the first workernode 114 as the new object owner 1120 of the object.

FIG. 11D is a block diagram illustrating an example of routing commandsto control plane controllers paired with worker nodes based upon whetherthe worker nodes are operational or have failed in accordance with anembodiment of the present technology. The distributed control plane 128may receive a subsequent command 1130 from an application hosted withinthe container orchestration platform 102. In some embodiments, thesubsequent command 1130 may be formatted according to the declarativemodel as attributes specified through the custom resource specificationof the custom resource definition within the distributed database 304 ofthe container orchestration platform 102 for the object. The subsequentcommand 1130 may target the same object that was targeted by the command1102 before the failure 1110 of the second worker node 116. Thesubsequent command 1130 may be received after the second worker node 116failed 1110 and after ownership of the object was reassigned to thefirst worker node 114.

The distributed control plane 128 may utilize the polling thread 901 todetermine ownership information 602 of the object targeted by thesubsequent command 1130. The ownership information 602 may indicate thatthe first worker node 114 is a current owner of the object targeted bythe subsequent command 1130. The distributed control plane 128 mayevaluate health status information of the first worker node 114 todetermine whether the first worker node 114 is operational (available toperform the subsequent command 1130) or has failed. In response todetermining that the first worker node 114 is operational, thedistributed control plane 128 may route the subsequent command 1130 tothe first control plane controller 136 paired with the first worker node114. The first control plane controller 136 may reformatted thesubsequent command 1130 to create a reformatted subsequent command 1132formatted according to the imperative model supported by the firstworker node 114. The first control plane controller 136 transmits thereformatted subsequent command 1132 to the first worker node 114 toexecute.

FIG. 12 is a sequence diagram illustrating operation of a control planecontroller in accordance with an embodiment of the present technology.During operation 1210, a client 1202 may generate and apply aspecification to a custom resource 1204. The specification may specifyvalues for parameters of the custom resource 1204, such as a size of avolume custom resource, a name of the volume custom resource, etc.During operation 1212, the client 1202 monitors the custom resource 1204for a status update regarding implementation of the specificationapplied to the custom resource 1204. During operation 1214, a controlplane controller 1206 monitors the custom resource 1204 and reacts tothe custom resource changes that occurred from the client 1202 applyingthe specification to the custom resource 1204. In some embodiments, thecustom resource 1204 is a resource that the control plane controller1206 reconciles and is node scoped/specific, and thus the control planecontroller 1206 may be run as a DaemonSet (e.g., a Kubernetes DaemonSetthat manages pods and ensures that pods are scheduled and running onnodes). The ownership of custom resources can change amongst nodes.Accordingly, the control plane controller 1206 implements logic thatfollows a common pattern of first deciding if this specific podassociated with the control plane controller 1206 should reconcile thecustom resource 1204 or not based upon whether the pod is paired with anode currently owning the custom resource 1204.

If the pod is paired with the node currently owning the custom resource1204, then operation 1216 is performed. During operation 1216, thecontrol plane controller 1206 evaluates the changes made to the customresource 1204 by the specification in order to determine what requeststo issue to an API endpoint 1208 of the node currently owning the customresource 1204 (e.g., a modify volume size request may be issued basedupon the specification modifying a size of a volume custom resource).During operation 1218, the control plane controller 1206 sends therequests to the API endpoint 1208 of the node currently owning thecustom resource 1204. During operation 1220, the node executes therequest (e.g., modifies the size of the volume custom resource). Duringoperation 1222, the node sends an API response through the API endpoint1208 to the control plane controller 1206. During operation 1224, thecontrol plane controller 1206 interprets the API response to identifyany custom resource modifications that were performed by the node whenexecuting the request. During operation 1226, the control planecontroller 1206 updates status fields of the custom resource 1204 basedupon the interpreted API response. During operation 1228, the client1202 identifies and/or reacts to the status updates within the customresource 1204 being monitored by the client 1202 for updates (e.g.,reacts to the size of the custom resource volume being increased such asby executing an operation that otherwise could not be executed if thesize was not increased).

FIG. 13 is a sequence diagram illustrating operation of a control planecontroller facilitating the creation a volume file in accordance with anembodiment of the present technology. During operation 1310, a client1302 may generate and apply a specification to a custom resource 1304 tocreate a volume file (a directory/dir1). The specification may specifyvalues for parameters for the volume file to create, such as a path(e.g., path:/dir1) and a type (e.g., type:/directory). During operation1312, the client 1302 monitors the custom resource 1304 for a statusupdate regarding implementation of the specification applied to thecustom resource 1304, such as a file operation success status conditionupdate. During operation 1314, a control plane controller 1306 monitorsthe custom resource 1304 and reacts to the custom resource changes thatoccurred from the client 1302 applying the specification to the customresource 1304. In some embodiments, the custom resource 1304 is aresource that the control plane controller 1306 reconciles and is nodescoped/specific, and thus the control plane controller 1306 may be runas a DaemonSet (e.g., a Kubernetes DaemonSet that manages pods andensures that pods are scheduled and running on nodes). The ownership ofcustom resources can change amongst nodes. Accordingly, the controlplane controller 1306 implements logic that follows a common pattern offirst deciding if this specific pod associated with the control planecontroller 1306 should reconcile the custom resource 1304 or not basedupon whether the pod is paired with a node currently owning the customresource 1304.

If the pod is paired with the node currently owning the custom resource1304 and determines that the volume file exists within the specificationfile collection (spec.files collection) and does not exist within astatus files collection (status.files collection) associated withalready created volume files, then operation 1316 is performed. Duringoperation 1316, the control plane controller 1306 creates a new filecreate parameters object to send through a request to the API endpoint1308 of the node currently owning the custom resource 1304. The new filecreate parameters object may comprise a volume UUID, the path specifiedby the specification (e.g., path:/dir1), and the type specified by thespecification (e.g., type:/directory). During operation 1318, thecontrol plane controller 1306 sends the requests to the API endpoint1308 of the node currently owning the custom resource 1304. Duringoperation 1320, the node executes the request by creating the newdirectory /dir1 at a root of the volume identified by the UUID. Duringoperation 1322, the node sends an API response through the API endpoint1308 to the control plane controller 1306. During operation 1324, thecontrol plane controller 1306 interprets the API response to determinethat the volume file (the directory /dir) was successfully recreated.The control plane controller 1306 updates the status files collection toindicate that the volume file (the directory /dir1) was created. Thecontrol plane controller 1306 also updates a file operation successstatus condition to indicate that all attempted requests weresuccessful. During operation 1326, the control plane controller 1306updates status fields of the custom resource 1304 based upon theinterpreted API response. During operation 1328, the client 1302identifies and/or reacts to the status updates within the customresource 1304 being monitored by the client 1302 for updates, such as byaccessing the new volume file (the directory /dir1).

FIG. 14 is a sequence diagram illustrating operation of a control planecontroller facilitating the creation of a volume snapshot in accordancewith an embodiment of the present technology. During operation 1410, aclient 1402 may generate and apply a specification to a custom resource1404 to create a volume snapshot. The specification may specify valuesfor parameters for the volume snapshot to create, such as a snapshotname, a volume name of the volume for which the snapshot is beingcreated, and a cluster identifier of a cluster owning the volume. Duringoperation 1412, the client 1402 monitors the custom resource 1404 for astatus update regarding implementation of the specification applied tothe custom resource 1404, such as a snapshot create success statuscondition update. During operation 1414, a control plane controller 1406monitors the custom resource 1404 and reacts to the custom resourcechanges that occurred from the client 1402 applying the specification tothe custom resource 1404. In some embodiments, the custom resource 1404is a resource that the control plane controller 1406 reconciles and isnode scoped/specific, and thus the control plane controller 1406 may berun as a DaemonSet (e.g., a Kubernetes DaemonSet that manages pods andensures that pods are scheduled and running on nodes). The ownership ofcustom resources can change amongst nodes. Accordingly, the controlplane controller 1406 implements logic that follows a common pattern offirst deciding if this specific pod associated with the control planecontroller 1406 should reconcile the custom resource 1404 or not basedupon whether the pod is paired with a node currently owning the customresource 1404. If the volume does not exist at the node, then the nodedoes not currently own the volume and a different control planecontroller paired with a different node actually owning the volume willinstead handle the creation of the snapshot.

If the pod is paired with the node currently owning the custom resource1404 (the volume being snapshotted), then operation 1416 is performed.During operation 1416, the control plane controller 1406 creates a newsnapshot create parameters object to send through a request to the APIendpoint 1408 of the node currently owning the custom resource 1404. Thenew snapshot create parameters object may comprise a volume UUID of thevolume and the snapshot name from the specification. During operation1418, the control plane controller 1406 sends the requests to the APIendpoint 1408 of the node currently owning the custom resource 1404.During operation 1420, the node executes the request by creating a newvolume snapshot of the volume at a root of the volume within a .snapdirectory and having the snapshot name. During operation 1422, the nodesends an API response through the API endpoint 1408 to the control planecontroller 1406. During operation 1424, the control plane controller1406 interprets the API response to determine that the volume snapshotwas successfully created. The control plane controller 1306 updates acreation time status of the volume snapshot, a ready to use status, anda restore size (e.g., a size needed to perform a restore operation usingthe volume snapshot) to reflect attributes of the newly created volumesnapshot. The control plane controller 1406 also updates a snapshotcreated success status condition to indicate that the attempted snapshotcreation operation was successful. During operation 1426, the controlplane controller 1406 updates status fields of the custom resource 1404based upon the interpreted API response. During operation 1428, theclient 1402 identifies and/or reacts to the status updates within thecustom resource 1404 being monitored by the client 1402 for updates.

FIG. 15 is an example of a computer readable medium in which variousembodiments of the present technology may be implemented. An exampleembodiment of a computer-readable medium or a computer-readable devicethat is devised in these ways is illustrated in FIG. 15 , wherein theimplementation comprises a computer-readable medium 1508, such as acompact disc-recordable (CD-R), a digital versatile disc-recordable(DVD-R), flash drive, a platter of a hard disk drive, etc., on which isencoded computer-readable data 1506. This computer-readable data 1506,such as binary data comprising at least one of a zero or a one, in turncomprises processor-executable computer instructions 1504 configured tooperate according to one or more of the principles set forth herein. Insome embodiments, the processor-executable computer instructions 1504are configured to perform at least some of the exemplary methods 1502disclosed herein, such as method 200 of FIG. 2 , method 500 of FIG. 5 ,method 700 of FIG. 7 , method 800 of FIG. 8 , and/or method 1000 of FIG.10 , for example. In some embodiments, the processor-executable computerinstructions 1504 are configured to implement a system, such as at leastsome of the exemplary systems disclosed herein, such as system 100 ofFIGS. 1A-1C, system 300 of FIG. 3 , system 600 of FIG. 6 , system 900 ofFIGS. 9A and 9B, and/or system 1100 of FIGS. 11A-11D, for example. Manysuch computer-readable media are contemplated to operate in accordancewith the techniques presented herein.

In an embodiment, the described methods and/or their equivalents may beimplemented with computer executable instructions. Thus, in anembodiment, a non-transitory computer readable/storage medium isconfigured with stored computer executable instructions of analgorithm/executable application that when executed by a machine(s)cause the machine(s) (and/or associated components) to perform themethod. Example machines include but are not limited to a processor, acomputer, a server operating in a cloud computing system, a serverconfigured in a Software as a Service (SaaS) architecture, a smartphone, and so on. In an embodiment, a computing device is implementedwith one or more executable algorithms that are configured to performany of the disclosed methods.

It will be appreciated that processes, architectures and/or proceduresdescribed herein can be implemented in hardware, firmware and/orsoftware. It will also be appreciated that the provisions set forthherein may apply to any type of special-purpose computer (e.g., filehost, storage server and/or storage serving appliance) and/orgeneral-purpose computer, including a standalone computer or portionthereof, embodied as or including a storage system. Moreover, theteachings herein can be configured to a variety of storage systemarchitectures including, but not limited to, a network-attached storageenvironment and/or a storage area network and disk assembly directlyattached to a client or host computer. Storage system should thereforebe taken broadly to include such arrangements in addition to anysubsystems configured to perform a storage function and associated withother equipment or systems.

In some embodiments, methods described and/or illustrated in thisdisclosure may be realized in whole or in part on computer-readablemedia. Computer readable media can include processor-executableinstructions configured to implement one or more of the methodspresented herein, and may include any mechanism for storing this datathat can be thereafter read by a computer system. Examples of computerreadable media include (hard) drives (e.g., accessible via networkattached storage (NAS)), Storage Area Networks (SAN), volatile andnon-volatile memory, such as read-only memory (ROM), random-accessmemory (RAM), electrically erasable programmable read-only memory(EEPROM) and/or flash memory, compact disk read only memory (CD-ROM)s,CD-Rs, compact disk re-writeable (CD-RW)s, DVDs, magnetic tape, opticalor non-optical data storage devices and/or any other medium which can beused to store data.

Some examples of the claimed subject matter have been described withreference to the drawings, where like reference numerals are generallyused to refer to like elements throughout. In the description, forpurposes of explanation, numerous specific details are set forth inorder to provide an understanding of the claimed subject matter. It maybe evident, however, that the claimed subject matter may be practicedwithout these specific details. Nothing in this detailed description isadmitted as prior art.

Although the subject matter has been described in language specific tostructural features or methodological acts, it is to be understood thatthe subject matter defined in the appended claims is not necessarilylimited to the specific features or acts described above. Rather, thespecific features and acts described above are disclosed as exampleforms of implementing at least some of the claims.

Various operations of embodiments are provided herein. The order inwhich some or all of the operations are described should not beconstrued to imply that these operations are necessarily orderdependent. Alternative ordering will be appreciated given the benefit ofthis description. Further, it will be understood that not all operationsare necessarily present in each embodiment provided herein. Also, itwill be understood that not all operations are necessary in someembodiments.

Furthermore, the claimed subject matter is implemented as a method,apparatus, or article of manufacture using standard application orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedsubject matter. The term “article of manufacture” as used herein isintended to encompass a computer application accessible from anycomputer-readable device, carrier, or media. Of course, manymodifications may be made to this configuration without departing fromthe scope or spirit of the claimed subject matter.

As used in this application, the terms “component”, “module,” “system”,“interface”, and the like are generally intended to refer to acomputer-related entity, either hardware, a combination of hardware andsoftware, software, or software in execution. For example, a componentincludes a process running on a processor, a processor, an object, anexecutable, a thread of execution, an application, or a computer. By wayof illustration, both an application running on a controller and thecontroller can be a component. One or more components residing within aprocess or thread of execution and a component may be localized on onecomputer or distributed between two or more computers.

Moreover, “exemplary” is used herein to mean serving as an example,instance, illustration, etc., and not necessarily as advantageous. Asused in this application, “or” is intended to mean an inclusive “or”rather than an exclusive “or”. In addition, “a” and “an” as used in thisapplication are generally be construed to mean “one or more” unlessspecified otherwise or clear from context to be directed to a singularform. Also, at least one of A and B and/or the like generally means A orB and/or both A and B. Furthermore, to the extent that “includes”,“having”, “has”, “with”, or variants thereof are used, such terms areintended to be inclusive in a manner similar to the term “comprising”.

Many modifications may be made to the instant disclosure withoutdeparting from the scope or spirit of the claimed subject matter. Unlessspecified otherwise, “first,” “second,” or the like are not intended toimply a temporal aspect, a spatial aspect, an ordering, etc. Rather,such terms are merely used as identifiers, names, etc. for features,elements, items, etc. For example, a first set of information and asecond set of information generally correspond to set of information Aand set of information B or two different or two identical sets ofinformation or the same set of information.

Also, although the disclosure has been shown and described with respectto one or more implementations, equivalent alterations and modificationswill occur to others skilled in the art based upon a reading andunderstanding of this specification and the annexed drawings. Thedisclosure includes all such modifications and alterations and islimited only by the scope of the following claims. In particular regardto the various functions performed by the above-described components(e.g., elements, resources, etc.), the terms used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., that is functionally equivalent), even though notstructurally equivalent to the disclosed structure. In addition, while aparticular feature of the disclosure may have been disclosed withrespect to only one of several implementations, such feature may becombined with one or more other features of the other implementations asmay be desired and advantageous for any given or particular application.

What is claimed is:
 1. A system, comprising: a distributed storagearchitecture including a plurality of worker nodes managing distributedstorage comprised of storage devices hosted by the plurality of workernodes; a container orchestration platform hosting applications runningthrough containers; and a distributed control plane hosted within thecontainer orchestration platform, wherein the distributed control planecomprises a plurality of pods hosting control plane controllers pairedwith the worker nodes, and wherein the distributed control plane selectsand routes a command to a control plane controller based upon thecommand targeting an object owned by a worker node paired with thecontrol plane controller, and wherein the control plane controller isconfigured to: receive the command, generated by an application runningas a container within the container orchestration platform, forperforming a control plane operation associated with an object, whereinthe command is formatted according to a first model supported by thecontainer orchestration platform; translate the command from beingformatted according to the first model into a reformatted commandformatted according to a second model supported by the distributedstorage architecture; and transmit the reformatted command through acall to an API endpoint of the worker node for implementing the controlplane operation through the distributed storage architecture.
 2. Thesystem of claim 1, wherein the distributed control plane is furtherconfigured to: in response to receiving a second command targeting theobject, select and route the second command to a second control planecontroller paired with a second worker node based upon a determinationthat the second worker node is a current owner of the object, whereinownership information maintained by the distributed storage architectureis queried using an identifier of the object to determine that theidentifier of the object is paired with an identifier of the secondworker node.
 3. The system of claim 1, the distributed control plane isfurther configured to: selectively route commands to control planecontrollers that are paired with worker nodes that are current owners ofobjects targeted by the commands, wherein the command is routed to thecontrol plane controller based upon the worker node being a currentowner of the object targeted by the command, wherein ownershipinformation maintained by the distributed storage architecture isqueried using an identifier of the object to determine that theidentifier of the object is paired with an identifier of the workernode.
 4. The system of claim 1, wherein the control plane operation isexecuted for a volume whose data is sliced across storage devices ofmultiple worker nodes of the distributed storage architecture, andwherein the worker node that executes the control plane operation is anowner of the volume tracked within ownership information maintained bythe distributed storage architecture.
 5. The system of claim 1, whereinthe control plane operation is executed to generate a snapshot of avolume owned by the worker node, wherein data of the volume is slicedacross storage devices of multiple worker nodes of the distributedstorage architecture, wherein ownership information is updated toindicate that the worker node is an owner of the snapshot.
 6. The systemof claim 1, wherein the control plane controllers are paired to theworker nodes according to a one to one relationship where a singlecontrol plane controller is paired with a single worker node, andwherein commands are routed to control plane controllers paired withworker nodes that are owners of objects targeted by the commands.
 7. Thesystem of claim 1, the distributed control plane is further configuredto: in response to determining that a new worker node has been added tothe distributed storage architecture, create and pair a new controlplane controller with the new worker node for communicating andformatting commands between the new worker node and the applicationswithin the container orchestration platform based upon the commandstargeting objects owned by the new worker node.
 8. The system of claim1, the distributed control plane is further configured to: in responseto determining that the worker node has been removed from thedistributed storage architecture, remove the control plane controllerfrom the container orchestration platform and transferring ownership ofobjects owned by the worker node to a different worker node.
 9. Thesystem of claim 1, the distributed control plane is further configuredto: process commands in parallel using multiple control planecontrollers based upon the commands targeting objects owned by differentworker nodes to which the control plane controllers are paired.
 10. Thesystem of claim 1, the distributed control plane is further configuredto: assign ownership of volume provisioning tasks to control planecontrollers based upon the control plane controllers being paired withworker nodes that are to own volumes being provisioned by the volumeprovisioning tasks.
 11. The system of claim 1, the distributed controlplane is further configured to: populate a custom resource definitionfor the object with information received within a response from theworker node, wherein the information comprises a status of the controlplane operation being implemented by the worker node.
 12. The system ofclaim 1, further comprising: a cluster master controller configured toimplement commands corresponding to infrastructure APIs including acreate cluster command, an add node command, and an add storage devicecommand.
 13. The system of claim 1, the distributed control plane isfurther configured to: reroute commands corresponding to volume APIs toa cluster master controller configured to create a hierarchy of objectsfor volumes created by the volume APIs.
 14. The system of claim 1,further comprising: a cluster master controller configured to preserve avolume core identifier of the object across the plurality of workernodes and corresponding control plane controllers.
 15. The system ofclaim 1, the control plane controller is further configured to: reformatcommands that are routed as reformatted commands to the worker nodebased upon the commands corresponding setting policies for objects ownedby the worker node, wherein the setting policies correspond to at leastone of a security policy, an export policy, or access control lists. 16.A method, comprising: retrieving, by a distributed control plane hostedwithin a container orchestration platform hosting applications runningthrough containers, a custom resource definition for an object storedwithin distributed storage comprised of storage devices hosted by aplurality of worker nodes of a distributed storage architecture;routing, by the distributed control plane, a command derived from thecustom resource definition to a control plane controller paired with aworker node based upon a determination that the worker node is an ownerof the object, wherein the command is formatted according to adeclarative model supported by the container orchestration platform;translating, by the control plane controller, the command from beingformatted according to the declarative model into a reformatted commandformatted according to an imperative model supported by the distributedstorage architecture; and transmitting, by the control plane controller,the reformatted command through a representational state transfer (REST)application programming interface (API) call to the worker node toimplement a control plane operation derived from the custom resourcedefinition.
 17. The method of claim 16, further comprising: routing, bythe distributed control plane, a second command to a second controlplane controller paired with a second worker node based upon the secondworker node being a new current owner of the object, wherein the newcurrent owner is identified as owning the object based upon ownershipinformation maintained by the distributed storage architecture.
 18. Themethod of claim 16, further comprising: tracking ownership of objects byworker nodes within ownership information; and routing commands tocontrol plane controllers that are paired with worker nodes that arecurrent owners of objects targeted by the commands based upon theownership information.
 19. The method of claim 16, further comprising:creating or removing control plane controllers of the distributedcontrol plane to scale the distributed control plane based upon a numberof worker nodes of the distributed storage architecture, wherein anumber of control plane controllers is maintained to match the number ofworker nodes.
 20. A non-transitory machine readable medium comprisinginstructions, which when executed by a machine, causes the machine to:evaluate a distributed storage architecture to identify a number ofworker nodes hosted by the distributed storage architecture, wherein theworker nodes host distributed storage comprised of storage deviceshosted across worker nodes; for each worker node in the distributedstorage architecture: create a pod within a container orchestrationplatform hosting applications running through containers; host a controlplane controller of a distributed control plane within the pod; pair thecontrol plane controller with a worker node according to a one to onepairing relationship; and configure the control plane controller tocommunicate and format commands between the worker node and theapplications based upon the worker node owning objects targeted by thecommands, wherein the objects are stored within the distributed storageof the distributed storage architecture; and control the distributedcontrol plane to selectively route commands to control plane controllersthat are paired with the worker nodes that are current owners of objectstargeted by the commands.